Use and Misuse of the Term “Experiment” in Mining Software Repositories Research

The significant momentum and importance of Mining Software Repositories (MSR) in Engineering (SE) has fostered new opportunities challenges for extensive empirical research. However, MSR researchers seem to struggle characterize the methods they use into existing SE body knowledge. This is especially case experiments. To provide evidence on special characteristics experiments their differences with traditionally acknowledged so far, we elicited hallmarks that differentiate an experiment from other types studies characterized MSR. We analyzed literature obtained a small-scale systematic mapping study assess term found 19% papers claiming be are indeed not at all but also observational studies, misleading way. From remaining 81% papers, only one them refers genuine controlled while others stand limited control. tend overlook such limitations, compromising interpretation results studies. recommendations insights support improvement